After switching the demo to the Franka arm + DH_PGI gripper, ik_interpolate can plan and physically lift the object, but the NMG planner still fails before replay. The observed failure is at the arm approach planning stage:
NeuralPlanner failed to plan arm trajectory for env 0.
PickUp failed to plan the approach trajectory.
This suggests the issue is not the gripper contact model anymore. The failure happens before physical grasp execution.
My current hypothesis is that this is mostly an out-of-distribution generalization issue for the current franka.pt NMG checkpoint. ik_interpolate solves each EEF waypoint geometrically against the live robot model/TCP, so it naturally adapts to the DH hybrid gripper and the selected grasp pose. NMG, however, is a learned waypoint policy. The hybrid DH gripper changes the effective TCP/tool geometry and produces grasp/approach poses that differ from the training distribution. Atomic pick-place also creates task-specific waypoints from grasp affordances, which are stricter than the planner-only FK/demo-offset benchmark.
Suggested improvements:

1. Add clearer NMG failure diagnostics: checkpoint path, start qpos, target waypoint pose, terminal pose error, active waypoint index, and whether rollout hit max_steps.
2. Try neural_refine as the default neural demo mode, or add IK snapping/projection for final key waypoints.
3. Rerank top-K grasp candidates by NMG reachability instead of selecting only the best antipodal/IK candidate.
4. Add fallback behavior: use NMG for free-space motion, but use IK for grasp approach/close/lift when NMG fails.
5. Fine-tune or retrain NMG on the Franka + DH gripper TCP distribution and atomic pick-place waypoints.
6. Include tool/TCP extrinsics in the NMG observation, or randomize TCP offsets during training to improve robustness to gripper changes.
7. Distill successful IK/TOPPRA pick-place trajectories into NMG training data.
8. Evaluate NMG separately on planner-only reachability and atomic pick-place, since the latter is much more sensitive to small TCP/pose errors.
   Uploading Kooha-2026-06-30-11-22-25.webm…